Capturing rich response relationships with small-data neural networks

ABSTRACT

The present disclosure relates to a response analysis system that employs a small-data training dataset to train a neural network that accurately performs domain-agnostic opinion mining. For example, in one or more embodiments, the response analysis system trains a response classification neural network using part of speech information (e.g., syntactic information) to learn and apply response classification labels for opinion text responses. In particular, the response analysis system employs part of speech information patterns without regard to word patterns to determine whether words in a text response correspond to an opinion, the target of the opinion, or neither. In addition, the trained response classification neural network has a significantly reduced learned parameter space, which decreases processing, memory requirements, and overall complexity.

BACKGROUND

Recent years have seen a significant increase in the use of computingdevices in creating, performing, and analyzing free-form opinion textresponses (e.g., survey responses, user feedback, and online reviews).Indeed, it is common for an individual to fill out a survey responseusing a computing device for a product or service offered by a business,person, school, or other entity. In many cases, a respondent provides atext response when expressing their opinion. For example, a respondentuses a mobile device to write a few sentences as part of a survey,social media post, or electronic message.

While conventional text analysis systems have attempted to extractfeedback information from text responses, the conventional systemsstruggle to accurately extract opinions from text responses. Indeed,conventional systems have employed the same feedback extractiontechniques and methods for over a decade. For example, most conventionalsystems employ application-level natural language processing to extractopinions from text responses. In particular, many conventional systemsemploy machine-learning algorithms in connection with natural languageprocessing for opinion mining. However, these conventional methodspresent several disadvantages, such as missing rich opinion insightsthat may be valuable to businesses, entities, and other entities.

One disadvantage, for instance, is that conventional systems requirelarge datasets to sufficiently train a machine-learning algorithm. Forexample, with English text responses, conventional systems need to learnproper context and meaning for about 200,000 unique words. To achievethis learning, a conventional system needs to learn tens of thousands ofsentences. Also, if the training dataset is not adequately large, themachine-learning algorithm will produce poor and inaccurate results.

As the size of a training dataset increases, additional problems arise.For example, a larger training dataset requires additional processingresources to obtain a learned parameter space as well as additionalmemory and storage to store the training dataset and the learned space.Indeed, the learned space is often too large to store on many clientdevices. Moreover, a larger training dataset requires additional rulesto learn, which in turn, increases the complexity and processing time ofexecuting the machine-learning algorithm in the future. Further, becauseconventional systems commonly learn using supervised training data, ahuman must manually tag, label, and/or annotate each sentence in thetraining dataset. The cost of labeling potentially hundreds of thousandsof items in large training dataset can be exorbitant and prohibitive.

As another problem, training datasets used by conventional systems areoften domain-specific. For instance, a conventional system attempting toanalyze text responses with respect to customer service must employ atraining dataset that includes words, terms, and phrases related tocustomer service. If the conventional system is mining opinions fromspecific or uncommon domains, a sufficiently large training dataset maynot exist or be available, which prevents the conventional system fromaccurately training a machine-learning algorithm.

In addition, conventional opinion mining systems often experienceout-of-vocabulary issues where the trained machine-learning algorithmencounters data not included in the training dataset. Thus, despite thelarge training dataset, the increased processing resources, and theadditional memory, conventional systems still produce inaccurateresults. Accordingly, these and other problems exist with regard toconventional systems and methods for opinion mining from free-form textresponses.

SUMMARY

Embodiments of the present disclosure provide benefits and/or solve oneor more of the foregoing or other problems in the art with systems,computer media, and methods for opinion mining from free-form opiniontext responses. For example, the disclosed systems employ uniquelyefficient small-data training datasets to accurately train a neuralnetwork to analyze, extract, classify, and provide filtered opinions tousers. In particular, the disclosed systems employ parts of speech(e.g., part of speech or syntactic information) and learned textresponse patterns to determine whether words and terms in a textresponse correspond to an opinion, the target of the opinion, or areirrelevant to the opinion (e.g., neither the opinion or target of theopinion).

For example, the disclosed systems can obtain a small-data trainingdataset that includes tagged sentences from text responses. For each ofthe sentences in the training dataset, the disclosed systems cangenerate one or more word-agnostic vectors based on a part of speechlabel and a response classification assigned to each word. Inparticular, the disclosed systems generate word-agnostic vectors that donot include word-semantic information, but rather other linguistic andsyntactical information corresponding to each word. Based on theword-agnostic vectors for each word in each sentence of the trainingdataset, the disclosed systems train a response classification neuralnetwork.

Moreover, the disclosed systems receive an input sentence of a textresponse that corresponds to an expressed opinion. In response, thedisclosed systems can determine a part of speech for each word in theinput sentence. Based on the parts of speech of the words, the disclosedsystems can employ the trained response classification neural networkmentioned above to determine a response classification label for eachword from the input sentence. Accordingly, the disclosed systems canpresent a target portion (e.g., words labeled as a target) and anopinion portion (e.g., words labeled as an opinion) of the inputsentence to a user.

The following description sets forth additional features and advantagesof one or more embodiments of the disclosed systems, computer media, andmethods. In some cases, such features and advantages will be obvious toa skilled artisan from the description or may be learned by the practiceof the disclosed embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description describes one or more embodiments withadditional specificity and detail through the use of the accompanyingdrawings, as briefly described below.

FIG. 1 illustrates a schematic diagram of an exemplary environment inwhich a response analysis system may be implemented in accordance withone or more embodiments.

FIG. 2A-2D illustrate example flow diagrams of training a responseclassification neural network to determine response classificationlabels in accordance with one or more embodiments.

FIG. 3 illustrates an example flow diagram of employing a trainedresponse classification neural network to determine responseclassification labels for an input sentence in accordance with one ormore embodiments.

FIGS. 4A and 4B illustrate example characteristic tables for a textresponse in accordance with one or more embodiments.

FIG. 5 illustrates an example flow diagram of employing multiple neuralnetworks to determine response classification labels for an inputsentence in accordance with one or more embodiments.

FIG. 6 illustrates an example graphical user interface for presentingfiltered opinion responses in accordance with one or more embodiments.

FIG. 7 illustrates a schematic diagram of the response analysis systemin accordance with one or more embodiments.

FIG. 8 illustrates a flowchart of a series of acts for training anddetermining response classification labels in accordance with one ormore embodiments.

FIG. 9 illustrates a block diagram of an exemplary computing device forimplementing one or more embodiments of the present disclosure.

FIG. 10 illustrates an example network environment of response analysissystem in accordance with one or more embodiments described herein.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of a response analysissystem that employs a small-data training dataset to train a neuralnetwork that accurately performs domain-agnostic opinion mining. Forexample, in one or more embodiments, the response analysis system trainsa response classification neural network using parts of speech (e.g.,part of speech or syntactic information) to learn and apply responseclassification labels for text responses that include an opinion of arespondent. As further described below, the response analysis systemtrains and employs a response classification neural network that has asignificantly reduced learned parameter space, which decreasesprocessing requirements, memory resources, and overall complexity of theresponse analysis system.

To illustrate, the response analysis system, in various embodiments,obtains a training dataset that includes words and sentences. In one ormore embodiments, the training dataset can be a small-data trainingdataset that includes a few thousand sentences (e.g., fewer than 2,500sentences). Notwithstanding the small-data training set, the responseanalysis system can use the principles described herein to train aneural network that results in a domain-agnostic neural network. Indeed,the domain of the small-data training set can correspond to any domain(e.g., topics such as sports, politics, engineering) and the resultingtrained response classification neural network is domain-agnostic (e.g.,can accurately and effectively processes text regardless of the domain).

As part of obtaining a small-data training dataset (or simply “trainingdataset”), the disclosed systems can generate word-agnostic vectorsbased on a part of speech label (e.g., a word's syntactic role in asentence) and a response classification assigned (e.g., manually-labeledor recursively generated) to each word of sentences within a trainingdataset. In general, a word-agnostic vector does not include semanticinformation of the word itself. Rather, the word-agnostic vectorcontains information corresponding to the word's part of speech as usedin a training or input sentence. In some embodiments, as part ofobtaining the small-data training dataset, the response analysis systemcan automatically determine and assign parts of speech to each of thewords in the training dataset of sentences to generates word-agnosticvectors based on identified parts of speech and response classificationlabels for each word in a sentence.

In some embodiments, a word-agnostic vector can contain other linguisticand syntactic information as well as information corresponding tosurrounding words in a sentence. In other words, the response analysissystem also can generate word-agnostic vectors based on additionalvector information. For example, the response analysis system can usedependency labels, attention mechanism information, and/or transferinformation to generate word-agnostic vectors.

Based on the word-agnostic vectors for words in each of the sentencesfrom the training dataset, in one or more embodiments, the responseanalysis system trains a response classification neural network. Forinstance, a trained response classification neural network indicatesdistances between parts of speech associated with word-agnostic vectorswithin the learned parameter space. Indeed, the response analysis systemuses the word-agnostic vectors corresponding to words in the trainingdataset to train the response classification neural network. In someembodiments, the response classification neural network is abi-directional long short-term memory (LSTM) recurrent neural network(RNN). Additionally, or alternatively, the response analysis systemtrains and employs additional neural networks. For example, the responseanalysis system can employ a second recurrent neural network to provideoutputted response classification labels back to the responseclassification neural network, as further described below.

Once the response analysis system trains a neural network with thetraining dataset, the response analysis system can employ the trainedneural network to determine response classification labels for inputtext that include opinions. For example, given an input sentence thatincludes input word(s), a trained response classification neural networkoutputs a response classification label for an input word within theinput sentence. Examples of response classification labels include anopinion classification label, a target classification label of theopinion (e.g., a subject of the opinion such as a person or product), ora label that indicates a word is neither a target or an opinion (e.g., aneither classification label).

To illustrate, in one or more embodiments, the response analysis systemreceives an input sentence that includes multiple words. Next, theresponse analysis system can determine parts of speech for each word inthe input sentence. Based on the parts of speech of the words, theresponse analysis system can employ the trained response classificationneural network to determine a response classification label for eachword from the input sentence (e.g., opinion, target, or neither). Bydetermining the response classification label for each word, theresponse analysis system can present a filtered version of the inputsentence to a user. For a given input sentence, the response analysissystem provides a target portion (e.g., words labeled as a target) andan opinion portion (e.g., words labeled as an opinion). In addition, theresponse analysis system can further analyze, classify, organize, and/orprocess filtered input sentences based on the classification labelscorresponding to words within the input sentence.

Although the response analysis system can analyze text from a variety ofsources, in one or more embodiments, the response analysis systemoperates in connection with a digital survey system. For instance, theresponse analysis system is part of, or communicates with, a digitalsurvey system that creates, administers, and collects survey responsesfrom electronic surveys (e.g., opinion text responses). For example, theresponse analysis system communicates with an administrator device(i.e., client devices) that enables an administrator to manage anelectronic survey. In addition, the response analysis system cancommunicate with respondent devices (i.e., client devices) wherebyrespondents provide opinion text responses in reply to electronic surveyquestions within the electronic survey.

Based on received opinion text responses, the response analysis systemcan determine response classification labels for each opinion textresponse using a trained response classification neural network. Forexample, using a trained response classification neural network, theresponse analysis system can provide an opinion text response as aninput to the classification neural network to determine a responseclassification label for each word in the opinion text response. Forinstance, the response analysis system receives an opinion textresponse, automatically identifies parts of speech—or word-agnosticvector information—for each word in the opinion text response. Theresponse analysis system can then employ the response classificationneural network to determine response classification labels for each wordin the opinion text response (e.g., a target, opinion, or neither).

Additionally, the response analysis system can filter opinion textresponses based on the response classification labels associated witheach word. For example, the response analysis system filters andpresents a target portion and an opinion portion to an administrator viaan administrator device. In this manner, the administrator can quicklyand accurately process filtered opinion text responses. In addition, theresponse analysis system can use the response classification labels tofurther organize, classify, and analyze opinion text responses.

The response analysis system provides many advantages and benefits overconventional systems and methods. For example, the response analysissystem overcomes the need for obtaining and employing large trainingdatasets along with their numerous accompanying issues. In particular,the response analysis system trains and operates a responseclassification neural network using word-agnostic vectors (e.g.,indicating parts of speech or syntactic information) rather than usingsemantic-specific word vectors (e.g., indicating semantic information ofunique words).

As mentioned above, and based on the word-agnostic vectors, the responseanalysis system can train a response classification neural network usinga small-data training dataset. For example, the response analysis systemcan accurately train the response classification neural network using1,000-2,500 sentences. In contrast, conventional systems require tens ofthousands or hundreds of thousands of sentences that are often domainspecific to particular subject matter. Accordingly, the cost and time ofpreparing and training a neural network with the small-data trainingdataset is trivial compared conventional systems.

In particular, the response analysis system learns recognized languagepatterns in connection with opinion text responses, as opposed toconventional system that learn based on semantic meanings for as manypossible words that could occur in an opinion text response.Accordingly, the response analysis system can employ a significantlysmaller training dataset using parts of speech (e.g., part of speechinformation) and achieve the same or more accurate results thanconventional systems.

To further illustrate, parts of speech are treated similarly across allsentences in a training dataset of opinion text responses and even innon-opinion training datasets. Indeed, the vast majority of sentenceswithin opinion text responses include a subject and a verb. Similarly,many opinion text responses include adverbs, adjectives, articles,objects, etc. Thus, because parts of speech are used in virtually everyopinion text response, significantly less data is needed to accuratelytrain the response classification neural network that is efficient,stable, and accurate. In contrast, conventional systems that train usingthe semantic meaning of words require a massive training dataset (e.g.,tens of thousands of sentences or more) to get an adequate sample sizefor each unique learned word, which further results in complex rules,system instability, and inaccurate results.

As another advantage, employing parts of speech as part of a trainingdataset enables the response analysis system to be domain-agnostic. Forexample, the response analysis system can train a neural network usingany type of opinion training dataset that has targets and opinionslabeled. For instance, using a training dataset related to Product X,the response analysis system can determine response classificationlabels for another product or service, as both the response for ProductX and the responses for another product or service include the samelanguage patterns with respect to parts of speech respect. In someembodiments, the response analysis system can train using a trainingdataset having non-opinion text responses that include objects (i.e.,targets) and language describing the objects (i.e., opinions).

Additionally, because employing parts of speech information enables theresponse analysis system to be domain-agnostic, the response analysissystem can identify response classification labels for responses forwhich no training dataset exists. Further, the response analysis systemcan use the same trained response classification neural network todetermine response classification labels for diverse subject areas(i.e., domains) with respect to opinion mining. Similarly, by employingparts of speech information to train and apply the responseclassification neural network, the response analysis system overcomesthe out-of-vocabulary issue. As mentioned above, a conventional systemmay encounter a word or phrase not found in the training dataset.However, since the response classification neural network isword-agnostic and the maximum number of possible parts of speech issmall (e.g., about 50), the out-of-vocabulary issue is minimized, if notaltogether eliminated.

The response analysis system also provides numerous computer processingand storage advantages. For example, because the number of possibleparts of speech is small and there are fewer rules to learn, theresulting learned parameter space of the response classification neuralnetwork is likewise small. Along with significantly reducing the storagespace needed to maintain the response classification neural network overconventional systems, the amount of computational processing resourcesand time to train and execute the response classification neural networkis also significantly reduced. Accordingly, most modern computingdevices can train, maintain, store, and employ the disclosed responseclassification neural network, as opposed to conventional system thatoften require specialized server systems to host large and complexneural networks.

Referring now to the figures, FIG. 1 illustrates a schematic diagram ofan environment 100 in which a response analysis system 104 may beimplemented in accordance with one or more embodiments. As illustrated,the environment 100 includes various computing devices including aserver device 101 that hosts an electronic survey system 102 (or simply“survey system”), a response analysis system 104, an administratorclient device 106 (associated with an administrator), and respondentclient devices 108 (associated with corresponding respondents). As shownin FIG. 1, each device is connected via a network 110. Additionaldetails regarding the various computing devices (e.g., the server device101, the administrator client device 106, and respondent client devices108) and networks (e.g., network 110) are explained below with respectto FIGS. 9 and 10.

As illustrated in FIG. 1, the environment 100 includes the server device101 that hosts the survey system 102 and the response analysis system104. In general, the survey system 102 facilitates the creation,administration, and collection of survey responses. For example, thesurvey system 102 enables an administrator, via the administrator clientdevice 106, to create, modify, and run a digital survey that includesfree-form opinion text responses. Similarly, the survey system 102provides electronic surveys to, and collects opinion text responsesfrom, respondents via the respondent client devices 108.

As shown, the survey system 102 includes the response analysis system104. For example, the response analysis system 104 analyzes opinion textresponses using a trained response classification neural network. Asmentioned above, the response analysis system 104 can train a responseclassification neural network using a small-data training dataset thatemploys word-agnostic vectors based on parts of speech and opinionlabels (i.e., response classification labels). Using the trainedresponse classification neural network, the response analysis system 104can determine response classification labels for portions of textresponses to identify targets and opinions corresponding to the textresponses.

In some embodiments, the response analysis system 104 is separate fromthe survey system 102. In one example, the response analysis system 104and the survey system 102 are separately located on the server device101. In another example, the response analysis system 104 is located ona separate server device than the survey system 102. In variousembodiments, the environment 100 includes the response analysis system104 and does not include the survey system 102. For example, theresponse analysis system 104 is located directly on the server device101 and determines response classification labels for opinion textresponses not obtained by the survey system 102.

Although FIG. 1 illustrates a particular arrangement of the serverdevice 101, administrator client device 106, respondent client devices108, and network 110, various additional arrangements are possible. Forexample, the administrator client device 106 can directly communicatewith the server device 101 hosting the response analysis system 104,bypassing the network 110. Further, while only one administrator clientdevice 106 is illustrated, the environment 100 can include any number ofadministrator client devices. Similarly, as shown, the environment 100can include any number of respondent client devices 108.

For reference, as used herein, the terms “opinion text response,” “textresponse,” or “response” refers to a string of electronic text. Forinstance, a text response is a free-form string of text that is providedin response to a prompt, query, question, survey, or experience. Often,a response includes one or more sentences of text (e.g., an inputsentence) provided by a respondent via a respondent client device. Forexample, a respondent answers a survey question regarding their opinionof a product or service they recently experienced (e.g., an opinion textresponse).

The term “training dataset” refers to a collection of sentences thathave been labeled with response classification labels. Often a trainingdataset is a small-data training dataset that includes only a fewthousand sentences (e.g., fewer than 2,500 sentences). The term“response classification label” refers to an opinion label assigned to aword. Examples of response classification labels include, but are notlimited to, a target classification label, an opinion classificationlabel, and a neither classification label (e.g., neither a target or anopinion). A “target” refers to an object of an opinion (e.g., a productor service). An “opinion” refers to description or sentiment expressedregarding the target. The term “neither” refers to words in a textresponse that are neither targets nor opinions. In some embodiments,based on the response classification labels determined for words in asentence, the sentence can be divided into various portions (e.g., atarget portion, an opinion portion, and a neither portion).

Words in a training dataset may be manually labeled. For example, ahuman may label words in the training dataset as either targets,opinions, or neither. In some embodiments, words that are neithertargets nor opinions are unlabeled. For words in unclassified responses(e.g., words in an input sentence), the response analysis system 104 candetermine corresponding response classification labels. In particular,the response analysis system 104 can employ a trained responseclassification neural network to determine response classificationlabels for words in a response.

The term “response classification neural network” refers to a neuralnetwork that can be, or is trained, to accurately determine responseclassification labels for text responses. In general, a neural networkis a subset of a machine-learning algorithm, which constructs andimplements algorithms that can learn from labeled training data and makepredictions on unlabeled input data. Accordingly, the responseclassification neural network may operate by building models fromexample inputs (e.g., training or learning), such as a training dataset,to make data-driven predictions or determinations. In some exampleembodiments, the response classification neural network includes abi-directional long short-term memory recurrent neural network.

The response analysis system 104 can train the response classificationneural network using word-agnostic vectors corresponding to the trainingdataset. As used herein, the term “word-agnostic vector” refers to aword vector that omits semantic word information corresponding to theword which it represents. A word-agnostic vector can include linguisticand syntactic information about a word other than semantic information,such as a part of speech information, dependency information, attentioninformation, and sequence information. In addition, a word-agnosticvector (or any word vector) is represented mathematically by numericalvalues, such as a vector or matrix of numbers.

As used herein, the term “part of speech” refers to a word's syntacticrole in a sentence. For example, the term “part of speech” refers to acategory to which a word is assigned in accordance with the word'ssyntactic functions. Largely, a part of speech indicates how a wordfunctions in meaning as well as grammatically within a sentence.Examples of parts of speech in English include, but are not limited tonoun, pronoun, verb, adverb, adjective, conjunction, preposition, andinterjection. Parts of speech can include additional and/or sub-parts ofspeech. For example, in English, there are around 50 different part ofspeech categories. While other languages may differ in the number ofclassified parts of speech (e.g., a number of languages employ twelveunique parts of speech), the total number is often relatively small(e.g., fewer than 100 parts of speech).

Turning now to FIGS. 2A-5, additional detail is provided regardingtraining and employing a response classification neural network. Toillustrate, FIGS. 2A-2D show example flow diagrams of training aresponse classification neural network to determine responseclassification labels. In particular, FIGS. 2A-2D show actions performedby the response analysis system 104 to train a response classificationneural network.

As illustrated in FIG. 2A, the response analysis system 104 obtainsresponses from a training dataset 202. In one or more embodiments, theresponse analysis system 104 accesses a database or storage locationthat includes responses that are labeled with response classificationlabels (e.g., target, opinion, neither). For example, the survey system102 and/or an administrator (e.g., via an administrator client device)provides the training dataset to the response analysis system 104. Insome embodiments, the response analysis system 104 obtains the trainingdataset from a remote location, such as a third-party computing device.

As mentioned previously, the training dataset can be a small-datatraining dataset that includes a few thousand sentences that are tagged,labeled, and/or annotated with response classification labels (e.g.,target, opinion, and/or neither). The training dataset can relate to anysubject matter that corresponds to respondents expressing their opinionin a text response. For example, the training dataset can be obtainedfrom a previous survey. Alternatively, the training dataset can bemanually generated or compiled by an administrator.

Upon obtaining the training dataset, the response analysis system 104generates 204 word-agnostic vectors for the words in the trainingdataset. As described above, a word-agnostic vector does not includesemantic word information, but includes other syntactic, sequence, andword pattern information. By ignoring words (i.e., not employing thesemantic meaning of words themselves), the response analysis system 104can train a response classification neural network using a much smallertraining dataset than conventional systems as well as avoid a host ofother problems, as detailed above.

As shown, generating word-agnostic vectors includes identifying 206parts of speech labels for each response. In one or more embodiments,the training dataset includes tags, labels, and/or annotations for thepart of speech assigned to each word. In some embodiments, the responseanalysis system 104 automatically determines the parts of speech of eachword for each sentence in the training dataset. For example, theresponse analysis system 104 employs a separate machine-learningalgorithm and/or process to identify the parts of speech associated witheach word. For instance, the response analysis system 104 inputs asentence from the training dataset into a machine-learning algorithm andreceives output indicating the part of speech associated with each word.

Using the part of speech assigned to a word, the response analysissystem 104 can construct a word-agnostic vector for the word. Forexample, the response analysis system 104 can uniquely indicate the partof speech assigned to a word using one or more numerical values. Forinstance, if the response analysis system 104 employs a neural network(e.g., a machine-learning algorithm) to determine parts of speech, theresponse analysis system 104 can use the latent representation outputfrom an intermediary layer or node of the neural network to form theword-agnostic vector for the word.

While the present disclosure describes one or more embodiments of theresponse analysis system 104 using English words and parts of speech,employing other languages are possible. For example, using the samemethods and techniques disclosed herein, the response analysis system104 can train and employ a response classification neural network in anylanguage that includes identifiable parts of speech, including languagesthat do not use the Roman alphabet.

As also shown in FIG. 2A, generating 204 word-agnostic vectors includesassigning 208 response classification labels to each response. Asmentioned above, words in the training dataset are associated with oneor more response classification labels, such as a target classificationlabel, an opinion classification label, or a neither classificationlabel. In one or more embodiments, the response analysis system 104converts (e.g., encodes for the purpose of training) the responseclassification label for a word to a numerical value such that thenumerical value can be incorporated into the word-agnostic vector forthe word. For instance, the response analysis system 104 uses a lookuptable to identify one or more numbers associated with a responseclassification label for a word and adds, multiplies, or otherwiseincorporates the one or more numbers to the word-agnostic vector thatindicates the part of speech for the word.

In other embodiments, the response analysis system 104 generates aseparate word-agnostic vector for each word in the training dataset. Forexample, for a particular word in a sentence of a response in thetraining dataset, the response analysis system 104 generates a firstword-agnostic vector that numerically indicates the word's part ofspeech and a second word-agnostic vector that numerically indicates theword's response classification label. In these embodiments, the responseanalysis system 104 inputs both word-agnostic vectors for the particularword to the response classification neural network as input to train theresponse classification neural network.

FIG. 2A illustrates the response analysis system 104 training 210 aresponse classification neural network. For instance, as just mentioned,the response analysis system 104 provides the generated word-agnosticvectors for each word in the training dataset as input to train theresponse classification neural network 210. As mentioned above, theresponse classification neural network uses word-agnostic vectors toconstructs logical connections that enable the classification neuralnetwork to make predictions on input data (e.g., an input sentence).

In one or more embodiments, the response classification neural networkis a bi-directional long short-term memory (LSTM) recurrent neuralnetwork (RNN). In some embodiments, the response classification neuralnetwork is another type of neural network or machine-learning algorithm.In many embodiments, the trained response classification neural networkindicates distances between parts of speech (or parts of speechpatterns) associated with two word-agnostic vectors. The shorter thedistances between two parts of speech (or parts of speech patterns) arewithin the trained multidimensional parameter space, the higher theprobability of a matching response classification label.

In various embodiments, the response analysis system 104 employssequence training while training the response classification neuralnetwork. For instance, sequence training includes using one or moreprevious words (i.e., word-agnostic vectors) in a sentence when learninglanguage patterns for subsequent words in the sentence. RNNs, inparticular, employ sequence learning to learn and recognize patterns insequences of data (i.e., terms and words).

As mentioned above, the response analysis system 104 employs asmall-data training dataset to train the response classification neuralnetwork. One reason why the response analysis system 104 can use atraining dataset that is significantly smaller than the training datasetof conventional systems is because the number of parts of speech isrelatively small. In the most general cases, the English languageincludes roughly 50 parts of speech or less. In any case, the totalnumber of parts of speech is minor compared to the learned data ofconventional systems (e.g., greater than 200,000 unique English words).

Because the number of parts of speech is small, the resulting learnedparameter space of a response classification neural network is alsosmall. Further, because the number of parts of speech is small, theresponse classification neural network needs very few rules to developthe learned parameter space. Accordingly, the complexity of the responseclassification neural network is greatly reduced and the computationaland storage requirements needed to train and employ the responseclassification neural network is also significantly reduced compared toconventional systems.

In additional embodiments, the response analysis system 104 can employ aloss function. For instance, the response analysis system 104 uses aloss function such as cross-entropy loss, based on the word-agnosticvectors to train a softmax classifier to classify word-agnostic vectorsusing the learned parameter space of the response classification neuralnetwork. Indeed, the softmax classifier is the final layer or node inthe response classification neural network that yields probabilityscores for each response classification label. In some embodiments, theresponse analysis system 104 inputs a determined response classificationlabel back into the response classification neural network for furthertraining, until the response classification neural network predictsaccurate response classification labels. In addition, the responseanalysis system 104 can perform other evaluations techniques to ensurethat a trained response classification neural network produces accurateresults.

Once the response analysis system 104 trains the response classificationneural network, the response analysis system 104 can use the trainedresponse classification neural network to determine labels for responsesthat include the opinions of respondents. In general, the responseanalysis system 104 can employ the trained response classificationneural network to determine response classification labels for futureresponses. Additional description regarding employing a trained responseclassification neural network is provided below with respect to FIG. 3.

FIGS. 2B-2D show additional embodiments employed by the responseanalysis system 104 to train the response classification neural network.To illustrate, in FIG. 2B, the response analysis system 104 generatesadditional vector information for each word in the training dataset. Forexample, in one or more embodiments, the response analysis system 104generates an additional word-agnostic vector for each word based ondependency information corresponding to the word. In some embodiments,the response analysis system 104 supplements an existing word-agnosticvector for a word with the additional dependency information.

As shown in FIG. 2B, the response analysis system 104 assigns 212dependency labels to each response in the training dataset. Dependencylabels can include a set of labels or annotations that indicate thestructure or hierarchy of words in a sentence. Similar to parts ofspeech, the response analysis system 104 can identify and/or generatedependency labels for words in a sentence that provide word-agnosticinformation about the word. Examples of dependency labels include, butare not limited to, root, subject, particle, determiner, adjectivepronoun, prepositional object, direct object, expletive, dependentclause, conjunct, coordinating conjunction, and separated verb prefix.

In some embodiments, the response analysis system 104 employs a separateneural network or other machine-learning algorithms to determine thedependency label for each word in the training dataset. Further, asdescribed above, the response analysis system 104 can convert thedependency label into one or more numerical values as part of generatingone or more word-agnostic vectors for each word.

As with parts of speech, the total number of dependency labels is alsorelatively small (e.g., approximately 50). Accordingly, adding additionword-agnostic vectors for each word corresponding to the word'sdependency label or supplementing the word's existing word-agnosticvector with the dependency label information does not significantlyimpact the size of the learned parameter space of the responseclassification neural network. For example, in the case that each wordhad a separate word-agnostic vector for a part of speech label and adependency label, the learned parameter space would roughly double insize (e.g., 50 parts of speech+50 dependencies) rather than multiply insize (e.g., 50 parts of speech×50 dependencies), which is stillsignificantly smaller than conventional systems.

As a note, because the response classification neural network employsword-agnostic vectors to recognize patterns, it is trivial that the sameword can have different parts of speech or different dependencies.Stated differently, the response classification neural network learnslanguage patterns, based on parts of speech and/or dependencies for asentence, that correspond to the response classification label assignedto the sentence. Thus, if the same word is used as a different part ofspeech in two different sentences in the training dataset, the responseanalysis system 104 ignores the semantics of the word and uses only thecorresponding part of speech (or other word-agnostic information) whentraining (and employing) the response classification neural network.

As shown in FIG. 2C, as part of generating 204 word-agnostic vectors,the response analysis system 104 also determines 214 attention mechanisminformation. In general, attention mechanism information providesinformation to the response classification neural network to learn thelikelihood of importance, for a word in a sentence, with respect toother words in the sentence. In this manner, the response classificationneural network can generate a weighted representation of each sentence,which can enable, in some cases, the response analysis system 104 toimprove performance and interpretability of the response classificationneural network.

The response analysis system 104 can employ attention mechanisminformation in one or more embodiments to help the responseclassification neural network properly connect words labeled with anopinion classification label with a corresponding target classificationlabel. In sentences where only one target classification label and oneopinion classification label are identified, the response classificationneural network can easily connect or pair the two labels together.However, longer sentences can include multiple target classificationlabels and multiple opinion classification labels. Thus, rather thanusing simple heuristics (e.g., proximity or chronological order), whichare often inaccurate, attention mechanism information can enable theresponse analysis system 104 to accurately disambiguate or separate aparticular target associated with a particular opinion. Additionaldescription regarding determining attention mechanism information foreach response is provided below in connection with FIG. 4B.

As also shown in FIG. 2C, as part of generating 204 word-agnosticvectors, the response analysis system 104 also identifies 216 transferinformation. Transfer information refers to vector information that isparticular to a set of responses. For instance, transfer informationincludes vector information for words that are customized, particularlyspecific, and/or highly peculiar. Examples transfer words include brandnames, words with embedded numbers, peculiar terms of art, and/or anentity's name. Transfer words can be tagged with a responseclassification label. In addition, transfer words can also be annotatedwith a part of speech label, dependency label, and/or otherword-agnostic vector information. As with the generated word-agnosticvectors, the response analysis system 104 can input word-agnosticvectors for transfer words into the response classification neuralnetwork to train the response classification neural network.

In some embodiments, in addition to word-agnostic information, transferwords can include semantic information. For example, the responseanalysis system 104 provides a separate word vector (e.g., a semanticword vector) to the response classification neural network thatsemantically identifies a transfer word along with its correspondingword-agnostic vector(s). In this manner, the response classificationneural network can train to recognize the specific set of transfer wordsalong with their corresponding word-agnostic information, which preventsthe response classification neural network from mislabeling transferwords in future responses.

While not shown, the response analysis system 104 can generate and/orprovide additional word-agnostic vector information as input to trainthe response classification neural network. For example, the responseanalysis system 104 provides double propagation labels for each word inthe training dataset. Further, in some embodiments, the responseanalysis system 104 also provides semantic word information for one ormore words in the training dataset.

FIG. 2D illustrates the response analysis system 104 training a responseclassification neural network based on multiple neural networks. Asshown, FIG. 2D includes the response classification neural networktrained in FIGS. 2A-2C. In addition, FIG. 2D shows employing 218 aclassified label feedback neural network (or simply feedback neuralnetwork) that provides response classification labels to the responseclassification neural network. In one or more embodiments, the feedbackneural network is a bi-directional long short-term memory recurrentneural network.

As shown, the response analysis system 104 employs the feedback neuralnetwork in connection with training the response classification neuralnetwork. For example, when the response classification neural networkdetermines a classification label for a word in a sentence, the feedbackneural network provides the determined classification label back to theresponse classification neural network when analyzing subsequent wordsin the same sentence. In this manner, the feedback neural networkensures that the response classification neural network identifies atleast one target classification label and one opinion classificationlabel for each sentence. Indeed, the feedback neural network preventshanging targets/opinions (e.g., sentences that include a targetclassification label without an opinion classification label, or viceversa). Additional description regarding the feedback neural network isprovided with respect to FIG. 5.

FIGS. 2A-2D illustrate various embodiments for generating word-agnosticvectors and training a response classification neural network. Whileparticular arrangements are shown, the response analysis system 104 canemploy alternative arrangements. For example, the response analysissystem 104 generates word-agnostic vectors using parts of speech labels,response classification labels, and transfer information. In anotherembodiment, the response analysis system 104 generates word-agnosticvectors using parts of speech labels, response classification labels,and attention mechanism information. Additionally, with each possiblecombination of generating word-agnostic vectors, the response analysissystem 104 can employ the feedback neural network and/or additionalneural networks.

FIG. 3, as mentioned above, illustrates a flow diagram of employing thetrained response classification neural network to determine responseclassification labels for an input sentence. For instance, the trainedresponse classification neural network can be the responseclassification neural network trained in connection with FIGS. 2A-2D.

As shown in FIG. 3, the response analysis system 104 receives 302 aninput sentence. For instance, the response analysis system 104 receivesa response from a survey system that includes one or more inputsentences where respondents express their opinions regarding a productservice, person, or any other topic. In other instances, the responseanalysis system 104 receives a batch of responses that each includes oneor more input sentences expressing opinions (e.g. survey responses). Foreach sentence in the response, the response analysis system 104 candetermine response classification labels, as detailed below.

Upon receiving the input sentence, the response analysis system 104identifies 304 parts of speech for the input sentence. For instance, theresponse analysis system 104 automatically analyzes the input sentenceto identify parts of speech for each word in the input sentence of aresponse, as described above. In some embodiments, the response analysissystem 104 employs a trained neural network or another trainedmachine-learning algorithm to identify various parts of speech in theinput sentence.

In one or more embodiments, the response analysis system 104 generates aword-agnostic vector for each word in the input sentence. For example,the response analysis system 104 employs the same actions, methods,and/or techniques, as described above with respect to generating aword-agnostic vector for sentences in the training dataset, to generatea word-agnostic vector for words in the input sentence. In this manner,the word-agnostic vectors are analogous between words and sentences theresponse analysis system uses to train the response classificationneural network and input words and input sentences.

In additional embodiments, the response analysis system 104 determinesadditional vector information for one or more words in the inputsentence. For example, the response analysis system 104 determines andincorporates dependency labels for words in the input sentence.Additionally, or alternatively, the response analysis system 104determines attention mechanism information and/or double propagationlabel information for words in the input sentence.

Upon identifying the parts of speech (e.g., word-agnostic vectors) forwords in the input sentence, the response analysis system 104 employsthe trained response classification neural network 306 to determine 308response classifications (i.e., response classification labels) for theinput sentence. In particular, the response analysis system 104 employsthe trained response classification neural network 306 to predict aresponse classification score for each word in the input sentence thatindicates a proper response classification label.

In general, the response analysis system 104 employs the sameword-agnostic vectors used to train the response classification neuralnetwork (e.g., training) as it the does for determining responseclassification labels of an input sentence using the trained responseclassification neural network. For example, the vector informationincluded in the word-agnostic vectors generated for training match tothe word-agnostic vector information generated for application on inputsentences. However, in some embodiments, the response analysis system104 employs different pieces of vector information between training andapplication. For instance, the response analysis system 104 generatesword-agnostic vectors that include all of the vector information shownin FIG. 2C for training, but generates word-agnostic vectors that onlyinclude parts of speech information for application, or vice versa.

Likewise, the response analysis system 104 can employ the secondfeedback neural network during application (i.e., when determiningresponse classification labels for words in the input sentence). In asimilar manner as described above, the feedback neural network canensure that the response classification neural network determines both atarget classification label and an opinion classification label for theinput sentence. Additional description regarding the second feedbackneural network is provided with respect to FIG. 5.

Further, as mentioned above, the trained response classification neuralnetwork 306 is domain-agnostic. In other words, even if the inputresponse corresponds to opinions for Product X and the trained responseclassification neural network 306 was trained based on opinionsresponses for Service Y, the trained response classification neuralnetwork 306 can still determine accurate response classification labelsbecause it is word-agnostic. In this manner, the response analysissystem 104 can employ the trained response classification neural network306 even if little or no training data exist for a particular product,service, or entity.

As also mentioned above, the trained response classification neuralnetwork 306 outputs response classification labels for one or more wordsin the input sentence. In one or more embodiments, the trained responseclassification neural network 306 only outputs words that have aclassification label of target or opinion. Similarly, the trainedresponse classification neural network 306 can ignore or omit words thatare labeled with a neither classification label. In some embodiments,the response analysis system 104 can provide the trained responseclassification neural network 306 with instructions (e.g., based on userinput) to selectively output (or ignore) words in the input sentencewith certain response classification labels.

As shown in FIG. 3, the response analysis system 104 filters 310responses based on the response classification labels determined foreach word in the input sentence. For example, the response analysissystem 104 filters the input sentence to include only words labeled withtarget classification labels and opinion classification labels. Indeed,the response analysis system 104 can filter the input sentence toinclude a target portion (e.g., words classified as targets) and anopinion portion (e.g., words classified as opinions) or other selectedportions. In addition, if the input sentence includes multiple pairs oftargets and opinions, the response analysis system 104 can separate thewords corresponding to each target/opinion pair.

Moreover, as shown, the response analysis system 104 presents 312 thefiltered sentence to a user, such as an administrator. For example, theresponse analysis system 104 presents the filtered sentence in theformat “target: opinion” to the user. In other examples, the responseanalysis system 104 provides a graphical user interface that enables theuser to see both a filtered version as well as an unfiltered version ofthe input sentence. Additional description regarding providing filteredresponses to a user (e.g., via an administrator device) is describedbelow with respect to FIG. 6.

In one or more embodiments, the response analysis system 104 alsoprovides one or more graphical elements that allow the user to providefeedback regarding the response classification labels. For example, theuser can approve the response classification labels provided by theresponse analysis system 104. Alternatively, the user providescorrections to one or more response classification labels (e.g.,modifies the response classification label from neither to opinion). Theresponse analysis system 104 can use the corrections to further trainand update the response classification neural network.

In addition to providing the filtered response to the user, in one ormore embodiments, the response analysis system 104 uses the responseswith the response classification labels to further analyze, classify,organize, and/or process the responses. For example, for each target,the response analysis system 104 determines statistics regardingpositive, negative, and neutral opinions. In another example, theresponse analysis system 104 ranks the most common opinions for one ormore targets. In other examples, the response analysis system 104arranges filtered responses based on target and/or opinion.

As described above, the response analysis system 104 trains a responseclassification neural network using sentences from a training dataset.In addition, the response analysis system 104 employs the trainedresponse classification neural network to determine responseclassification labels for input sentences. To further describe how theresponse analysis system 104 both trains and employs a responseclassification neural network, FIGS. 4A and 4B illustrate characteristictables for a text response.

As shown, FIG. 4A includes a first response characteristics table 410and FIG. 4B includes a second response characteristics table 420. Boththe first response characteristics table 410 and the second responsecharacteristics table 420 include the response 412, 422 with the words,“Acme had really good service.”

As shown in FIG. 4A, the first response characteristics table 410includes a parts of speech label 414 and a response classification label416 for each word of the response 412. Based on the response, theresponse analysis system 104 can identify the parts of speech label 414for each word in the response, as described above. If the response 412was part of the training dataset used to train a response classificationneural network, the response analysis system 104 identifies the responseclassification label 416 for each word from the training dataset.Alternatively, if the response 412 was from an input sentence, theresponse analysis system 104 uses the trained response classificationneural network to determine the response classification label for eachword, as described above.

To illustrate, suppose the response analysis system 104 employed theresponse 412 to train a response classification neural network.Accordingly, the response analysis system 104 obtains the parts ofspeech labels and response classification labels for each word in theresponse 412 and generates word-agnostic vectors for each word in theresponse 412. Additionally, the response analysis system 104 ignores thewords themselves of the response 412. Accordingly, the response analysissystem 104 generates word-agnostic vectors that represent the pattern:proper noun/neither, verb/neither, adverb/opinion, adjective/opinion,and noun/target. Using the combinations of parts of speech labels 414and response classification labels 416 (and any other provided vectorinformation), the response analysis system 104 trains the responseclassification neural network, as described above.

Now suppose the response 412 is an input sentence from an opinion textresponse. As mentioned above, the response analysis system 104identifies the parts of speech labels for each word in the sentence(e.g., in the form of a word-agnostic vector). The response analysissystem 104 again ignores the words and provides the parts of speechlabels to the trained response classification neural network.Accordingly, the response analysis system 104 inputs the response to thetrained response classification neural network using the pattern: propernoun, verb, adverb, adjective, and noun.

The trained response classification neural network processes the inputpattern to determine scores that indicate a response classificationlabel for each word of the response 412 (e.g., using a softmaxclassifier loss function). In particular, based on the determinedscores, the response analysis system 104 employs the trained responseclassification neural network to output classification labels of:neither, neither, opinion, opinion, target. The response analysis system104 then associates the output with each corresponding word in theresponse 412.

As mentioned above, in some embodiments, the response analysis system104 filters responses to a target portion and an opinion portion. Toillustrate, the response analysis system 104 can filter the response 412as target: “services” and opinion: “really good.” As another example,the response analysis system 104 can filter the response 412 using the“target: services” format as “services: really good.” Additionaldescription regarding providing filtered responses to a user isdescribed below with respect to FIG. 6.

FIG. 4B, as mentioned above, provides an example of the responseanalysis system 104 determining attention mechanism information for asentence of a response. As shown, the second response characteristicstable 420 includes the response 422 having the words of “Acme had reallygood service.” In addition, the second response characteristics table420 includes a most attended word 424. The response analysis system 104can determine the most attended word 424 for each word in the response422. As with FIG. 4A, the response analysis system 104 can determine themost attended word 424 to a sentence in the training dataset or an inputsentence.

As described above, the response analysis system 104 can provideattention mechanism information (e.g., a numerical indication of themost attended word 424) to the response classification neural networkwithin a word-agnostic vector. As also described above, attentionmechanism information can assist a response classification neuralnetwork in disambiguating or separating connected targets and opinionswhen a sentence includes multiple targets and multiple opinions.

In one or more embodiments, the response analysis system 104 determinesthe most attended word 424 for a word in the response by employing anattention mechanism matrix. For instance, the response analysis system104 generates or otherwise obtains the attention mechanism matrix. Insome embodiments, the response analysis system 104 trains an attentionmechanism matrix to map opinions to targets. In this manner, theattention mechanism matrix provides weights that not only indicate whatother word in a sentence the response classification neural networkshould focus on given an input word (e.g., the most attended word 424for each input word), but also the likelihood that the words form atarget that connects to an opinion (e.g., a target and opinion pair).

Using the trained or otherwise obtained attention mechanism matrix, theresponse analysis system 104 calculates weighted scores for each pair ofwords in an input sentence. For example, the response analysis system104 multiplies the word vectors of each word in the pair by theattention mechanism matrix to obtain a weighted score (e.g., Word Vector1×Attention Mechanism Matrix×Word Vector 2=Weight). The responseanalysis system 104 can then apply the weighted score to one or morewords in the word pair.

To illustrate, given the input word “really” in the response 422, theresponse analysis system 104 first identifies each word pairing (e.g.,really: Acme, really: had, really: good, and really: services). For eachword pairing, the response analysis system 104 applies the attentionmechanism matrix to the corresponding word vectors. Based on theweighted scores, the response analysis system 104 determines that themost attended word 424 for the word “really” is “services.” In someembodiments, as mentioned above, the most attended word pairing canindicate a connection between a target word and an opinion word (e.g.,“really” is an opinion and “service” is the target). The responseanalysis system 104 can repeat the process to determine the mostattended word 424 for each word in the response 422.

As mentioned above, the response analysis system 104 can also apply thecorresponding weighted score based on the most attended word 424. Forinstance, in one or more embodiments, the response analysis system 104weights one or more of the word-agnostic vectors for the word “really”based on the weighted score (e.g., a score between 0-1) with thecorresponding most attended word 424 of “services.” In otherembodiments, the response analysis system 104 weights one or more of theword-agnostic vectors for the word “really” based on each weighted scorebetween the word “really” and each other word in the response 422.

By employing the attention mechanism matrix, the response analysissystem 104 can achieve a weighted representation for words in theresponse 422. Further, the response analysis system 104 can input theweighted representation of the response 422 (e.g., attention mechanisminformation for each word in the response 422) to the responseclassification neural network to enable the response classificationneural network to better disambiguate target/opinion pairs when asentence includes multiple target/opinion pairs.

FIG. 5 illustrates a flow diagram 500 of employing multiple neuralnetworks to determine response classification labels for an inputsentence in accordance with one or more embodiments. As mentioned above,the response analysis system 104 can employ the response classificationneural network and an additional neural network, such as a feedbackneural network, when determining response classification labels forwords in a sentence. Indeed, the response analysis system 104 can employmultiple neural networks either during training and/or when determiningresponse classification labels for input sentences.

As also mentioned above, in one or more embodiments, the responseanalysis system 104 employs multiple neural networks to prevent thecreation of hanging targets/opinions in a sentence (e.g., a sentencethat includes a target classification label without an opinionclassification label, or vice versa). In rare instances, a single neuralnetwork may not factor in determinations (e.g., determined responseclassification labels) for previous words in the sentence, which canlead to the neural network not identifying both a target and acorresponding opinion for the sentence.

To illustrate, FIG. 5 includes an input sentence 502 of “the managerlied.” In this sentence, the word “lied” is a verb. Because it isuncommon for verbs to be wholesale opinions, the response classificationneural network may output the response classification labels for theinput sentence 502 as: neither, target, neither. In this case, thesentence has a hanging target classification label with no correspondingopinion.

To remedy this issue, FIG. 5 also shows a prediction long short-termmemory recurrent neural network 504 (or simply “prediction neuralnetwork 504”) and an encoded classification feedback long short-termmemory recurrent neural network 506 (or simply “encoded feedback neuralnetwork 506”). The prediction neural network 504 can be one or moreembodiments of the response classification neural network describedabove. Similarly, the encoded feedback neural network 506 can be one ormore embodiments of the feedback neural network described above.

The response analysis system 104 can provide the input sentence 502(e.g., parts of speech corresponding to words in the sentence) to theprediction neural network 504, as described above. For the first word inthe input sentence 502 (i.e., “the”) the prediction neural network 504determines the response classification label of neither. For example,the prediction neural network 504 has been trained to recognize “the”(e.g., definite articles) or “a” (e.g., indefinite articles) at thebeginning of a sentence as neither a target nor an opinion.

As shown, the prediction neural network 504 outputs 508 a the responseclassification label encoded as “neither” for the word “the” to theencoded feedback neural network 506. The encoded feedback neural network506 receives the response classification label as input and provides 508b the response classification label back to the prediction neuralnetwork 504. In particular, the encoded feedback neural network 506provides a corresponding encoder latent vector for the word thatindicates the encoder's state (e.g., the word's forwards and backwardscontext) back to the prediction neural network 504. In some embodiments,the encoded feedback neural network 506 only outputs a responseclassification label when the classification label is either a target oran opinion.

The prediction neural network 504 then determines a responseclassification label for the word of “manager.” As part of determiningthe response classification labels, the prediction neural network 504factors in the response classification labels determined for previouswords in the sentence (e.g., based on the provided encoder statementioned above). In particular, the prediction neural network 504 islooking for either a target classification label or an opinionclassification label, such that the prediction neural network 504 cancomplete the target and opinion pair. In this case, the neitherclassification label does not affect the response classificationdetermination for the word “manager,” which the prediction neuralnetwork 504 determines to be a target.

Again, the prediction neural network 504 outputs 508 c the responseclassification label encoded as “target” for the word “manager” to theencoded feedback neural network 506. The encoded feedback neural network506 receives the response classification label as input and provides 508d the response classification label back to the prediction neuralnetwork 504. When the prediction neural network 504 encodes the lastword (i.e., “lied”) in the input sentence 502, the prediction neuralnetwork 504 recognizes that the sentence includes a target, but noopinion. Accordingly, the prediction neural network 504 determines ifthe last word (e.g., based on the word's part of speech) in the inputsentence 502 could serve as an opinion.

As shown, the prediction neural network 504 determines and outputs 508 ethe response classification labels encoded as “opinion,” along with theother response classification labels 510 for the input sentence. In thismanner, the response analysis system 104 employs both the predictionneural network 504 and the encoded feedback neural network 506 to ensurea pairing between a target and an opinion. Indeed, employing themultiple neural networks, as described herein, reduces sentences withhanging classification labels from about 10% to (e.g., 40-70 per 1,000sentences) less than 1% (2-5 per 1,000 sentence) based the inventor'sanalysis of the response analysis system employing the multiple neuralnetworks as described above.

Turning now to FIG. 6, additional description is provided regardingpresenting filtered responses to a user, as mentioned above. Inparticular, FIG. 6 illustrates a graphical user interface for presentingfiltered opinion responses. As shown, FIG. 6 includes a client device600 that displays a graphical user interface 602. In one or moreembodiments, the response analysis system 104 provides the graphicaluser interface 602, which can be displayed to an administrator via anadministrator client device. In some embodiments, as describedpreviously, the response analysis system 104 is part of a survey systemthat facilitates the creation, administration, and collection ofsurveys.

As shown, the graphical user interface 602 includes a prompt or surveyquestion 604 given to multiple respondents. The responses shown in thegraphical user interface 602 correspond to the survey question 604(e.g., feedback at a car dealership). For instance, the graphical userinterface 602 includes a list of filtered responses 606 and acorresponding list of full responses 608 (i.e., non-filtered responses).

The filtered responses 606 employ the format of “target: opinion.” Inother words, each response in the list of filtered responses 606displays a target portion and a corresponding opinion portion. In thismanner, an administrator can quickly, clearly, and easily assess therelevance of a respondent's opinion without needing to read the entireresponse. However, if the administrator needs to read the entireresponse, the full response is provided adjacent to the filteredresponse in the list of full responses 608.

To illustrate, the third filtered responses 608 a reads “manager: dobetter job,” which is quicker to read and coveys the same message as thefull response. As another example, the fifth filtered responses 606 bidentifies three pairs of targets and opinions. In this example, theresponse analysis system 104 presents each target/opinion pairseparately, which enables the administrator to extract valuableinformation from the response much more quickly and effortlessly thanhaving to read the full response, manually identify the multiple targetsand opinions, and mentally separate each pair. Additionally, theresponse analysis system 104 can treat each target/opinion pairseparately when processing, analyzing, and grouping opinions, asdescribed below.

In one or more embodiments, the response analysis system 104 providesadditional graphical user interfaces to an administrator. For example,the response analysis system 104 provides a list of identified targetsfor one or more survey questions 604. The response analysis system 104can sort the list of targets by alphabetical order, chronological order,manually ordered, or by any number of ordering criteria. In addition,the response analysis system 104 can list each of the opinions thatcorrespond to the target adjacent to the target, or vice versa. Toillustrate, for the target of “manager,” the response analysis system104 lists the opinions of “do better job” and “very helpful.”

In some embodiments, the response analysis system 104 provides agraphical user interface that sorts the responses by positive, neutral,and negative opinions. In addition, the response analysis system 104 candetermine statistics regarding positive, negative, and neutral opinions.In other examples, the response analysis system 104 ranks the mostcommon opinions for one or more targets. Further, the response analysissystem 104 automatically marks the responses with negative and/orneutral opinions for a customer service representative to follow up.

In various embodiments, as mentioned above, the response analysis system104 provides one or more graphical elements (e.g., buttons, text fields,drop-down menus, etc.) that allow an administrator to provide feedbackregarding the response classification labels. In one example, theadministrator can approve or disapprove of the response classificationlabels for a response. In other examples, the administrator reclassifiesa word that is incorrectly classified. Upon detecting the change, theresponse analysis system 104 can update the filtered responses 606within the graphical user interface to reflect the change. In addition,the response analysis system 104 can use the corrections to furthertrain and update the response classification neural network.

Referring now to FIG. 7, additional detail will be provided regardingcapabilities and components of the response analysis system 104 inaccordance with one or more embodiments. In particular, FIG. 7 shows aschematic diagram of a response analysis system 104 hosted on acomputing device 700. The response analysis system 104 can represent oneor more embodiments previously described. For example, while not shown,in some embodiments, the response analysis system 104 is implemented ina survey system (e.g., the survey system 102).

As shown, the response analysis system 104 is located on a computingdevice 700. In general, the computing device 700 may represent varioustypes of client devices, such as a mobile client device (e.g., a mobiletelephone, a smartphone, a PDA, a tablet, a laptop, etc.). In otherembodiments, the computing device 700 is a non-mobile device, such as adesktop or server, or another type of client device. Additional detailswith regard to the computing device 700 are discussed below as well aswith respect to FIG. 9.

As illustrated in FIG. 7, the response analysis system 104 includesvarious components. For example, the response analysis system 104includes a response classification trainer 704 and a response manager706. The response classification trainer 704 includes a word-agnosticvector generator 708 having a parts of speech identifier 710 andadditional vector information identifier 712, a response classificationneural network 714, and a training database 716 storing parts of speechlabels 718 and response classification labels 720. The response manager706 includes a response classifier 722, a response report manager 724,and a response database 726 storing text responses 728 and responseclassification data 730.

The response classification trainer 704, in general, trains one or moreneural network and/or machine-learning algorithms for the purpose ofdetermining or predicting response classification labels for words inopinion responses. For example, in one or more embodiments, the responseclassification trainer 704 trains a response classification neuralnetwork 714, as previously described. In addition, the responseclassification trainer 704 can generate additional neural networks ormachine-learning algorithms, such as a feedback neural network, a partsof speech determination neural network, an attention mechanism neuralnetwork, and a word/sentence dependency neural network, each of which isdescribed above.

As shown, the response classification trainer 704 includes theword-agnostic vector generator 708. In general, the word-agnostic vectorgenerator 708 generates one or more word-agnostic vectors for each wordinput into the response classification neural network, either fortraining or for prediction. For example, the word-agnostic vectorgenerator 708 generates word-agnostic vectors for words in a trainingdataset. In another example, the word-agnostic vector generator 708 cangenerate word-agnostic vectors for words in an input sentence, asdescribed below.

In one or more embodiments, the word-agnostic vector generator 708generates a word-agnostic vector for words based on the word's parts ofspeech. For example, the word-agnostic vector generator 708 accesses theparts of speech labels 718 and the response classification labels 720stored in the training database 716, or from another source, aspreviously described. In various embodiments, the word-agnostic vectorgenerator 708 converts the parts of speech labels 718 and/or theresponse classification labels 720 for each word in a training sentenceinto one or more word-agnostic vectors.

In some embodiments, the word-agnostic vector generator 708 determinesthe parts of speech label for a word. In these embodiments, the parts ofspeech identifier 710 within the word-agnostic vector generator 708identifies a word's parts of speech. In general, the parts of speechidentifier 710 parses a sentence to determine the parts of speech foreach word in the sentence. In addition, the parts of speech identifier710 can label, tag, or otherwise annotate each word with its determinedpart of speech. As described above, the parts of speech identifier 710can employ a neural network and/or a machine-learning algorithm toautomatically determine the parts of speech for words in a sentence.

In additional embodiments, the word-agnostic vector generator 708generates further word-agnostic vectors (or incorporates addedinformation into existing word-agnostic vectors) for a word based onadditional vector information for the word. For example, the additionalvector information identifier 712 obtains and/or generates additionalvector information, such as dependency label, attention mechanisminformation, and/or double propagation label information as describedabove. In various embodiments, the additional vector informationidentifier 712 generates a word-agnostic vector based on transferinformation, as previously described. In some embodiments, theadditional vector information identifier 712 generates one or moresemantic word vectors for transfer words and/or other words in thetraining dataset.

Using the word-agnostic vectors and/or additional word vectors (e.g.,semantic word vectors), the response classification trainer 704 trainsthe response classification neural network 714. As described above indetail, the response classification trainer 704 can provideword-agnostic vectors that indicate parts of speech labels 718, responseclassification labels 720, and optionally additional vector informationas input to train the response classification neural network toaccurately predict response classification labels for words in an inputsentence.

In one or more embodiments, the response classification neural network714 is a learned parameter space (e.g., latent or non-latent) that isused to predict response classification labels based on detected partsof speech patterns. For example, in some embodiments, the trainedresponse classification neural network 714 indicates the distancesbetween parts of speech associated with two word-agnostic vectors inmultidimensional vector space. For instance, the shorter (or fartherdepending on the implantation) the distances between two parts of speechare within the trained parameter space, the higher the probability of amatching response classification label. Because the trained parameterspace is multidimensional, additional considerations, such as parts ofspeech surrounding the port of speech of an input word also affect theresponse classification label determination.

The training database 716, as mentioned above, includes the parts ofspeech labels 718 and the response classification labels 720. In someembodiments, the parts of speech labels 718 and response classificationlabels 720 correspond to responses (i.e., terms, words, and sentences)in a training dataset. As mentioned above, in many embodiments, thetraining dataset is a small-data training dataset that includes arelatively small number of sentences for training purposes.

The response manager 706 can store, receive, process, analyze, and/ororganize responses (e.g., text responses 728) within the computingdevice 700. For example, in one or more embodiments, the responsemanager 706 identifies one or more unclassified opinion text responses(e.g., survey responses). In various embodiments, the response manager706 stores and retrieves text responses 728 from the response database726.

The response manager 706 can input the responses into the trainedresponse classification neural network 714 using the response classifier722. For instance, the response classifier 722 provides word-agnosticvectors that represent each word in an input sentence to the trainedresponse classification neural network 714. In some embodiments, theresponse classifier 722 obtains one or more word-agnostic vectors forwords of an input sentence. For instance, the response classifier 722communicates with the word-agnostic vector generator 708 described aboveto generate one or more word-agnostic vectors. Alternatively, theresponse classifier 722 generates or otherwise obtains word-agnosticvectors for words in an input sentence.

Upon the trained response classification neural network 714 outputting adetermination (i.e., response classification labels) for one or morewords in the input sentence, the response classifier 722 can provide theresponse classification labels to the response database 726 as responseclassification data 730 in connection with the text responses 728. Inaddition, the response classifier 722 can provide the responseclassification labels to the response report manager 724.

The response report manager 724 can provide classified responses to auser, such as an administrator. For example, the response report manager724 provides a graphical user interface that presents filtered responsesthat include a target portion and an opinion portion, as described aboveand shown in FIG. 6. In addition, the response report manager 724 canfurther analyze, organize, process, and/or classify the classifiedresponse. In some embodiments, the response report manager 724 accessesfull responses and filtered responses from the response database 726, aspreviously described.

Each of the components 704-730 of the response analysis system 104 caninclude software, hardware, or both. For example, the components 704-730can include one or more instructions stored on a computer-readablestorage medium and executable by processors of one or more computingdevices, such as a client device or server device. When executed by theone or more processors, the computer-executable instructions of theresponse analysis system 104 can cause the computing device(s) toperform the feature learning methods described herein. Alternatively,the components 704-730 can include hardware, such as a special-purposeprocessing device to perform a certain function or group of functions.Alternatively, the components 704-730 of the response analysis system104 can include a combination of computer-executable instructions andhardware.

Furthermore, the components 704-730 of the response analysis system 104may, for example, be implemented as one or more operating systems, asone or more stand-alone applications, as one or more modules of anapplication, as one or more plug-ins, as one or more library functionsor functions that may be called by other applications, and/or as acloud-computing model. Thus, the components 704-730 may be implementedas a stand-alone application, such as a desktop or mobile application.Furthermore, the components 704-730 may be implemented as one or moreweb-based applications hosted on a remote server. The components 704-730may also be implemented in a suite of mobile device applications or“apps.”

FIGS. 1-7, the corresponding text, and the examples provide a number ofdifferent methods, systems, devices, and non-transitorycomputer-readable media of the response analysis system 104. In additionto the foregoing, one or more embodiments can also be described in termsof flowcharts comprising acts for accomplishing a particular result. Forexample, FIG. 8 illustrates a flowchart of a series of acts 800 oftraining and determining response classification labels in accordancewith one or more embodiments.

While FIG. 8 illustrates acts according to one embodiment, alternativeembodiments may omit, add to, reorder, and/or modify any of the actsshown in FIG. 8. The acts of FIG. 8 can be performed as part of amethod. Alternatively, a non-transitory computer-readable medium cancomprise instructions that, when executed by one or more processors,cause a computing device to perform the acts of FIG. 8. In someembodiments, a system can perform the acts of FIG. 8. In addition, inone or more embodiments, the series of acts 800 is implemented on one ormore computing devices, such as a server device 101 and/or computingdevice 700.

The series of acts 800 includes an act 810 of receiving sentences thatinclude multiple words. In particular, the act 810 can involve receivinga plurality of sentences that comprise a plurality of words. In one ormore embodiments, the act 810 includes obtaining a training datasetincluding a plurality of sentences that include a plurality of words,where each of the plurality of words is assigned a responseclassification label. In some embodiments, the training dataset is asmall-data training dataset that includes 2,500 or fewer sentences.

As shown, the series of acts 800 also includes an act 820 of generatingword-agnostic vectors for the words based on parts of speech andresponse classifications. In particular, the act 820 can involvegenerating word-agnostic vectors for the plurality of words from each ofthe plurality of sentences based on a part of speech label assigned toeach word of the plurality of words and a response classification labelassigned to each word of the plurality of words. In one or moreembodiments, the act 820 also includes generating the word-agnosticvectors based on a dependency label assigned to each word of theplurality of words and/or transfer information that classifies one ormore particular words not included in the training dataset with anassigned part of speech label. In some embodiments, the word-agnosticvectors omit semantic word information corresponding to the plurality ofwords from each of the plurality of sentences.

As shown in FIG. 8, the series of acts 800 further includes an act 830of training a response classification neural network based on theword-agnostic vectors. In particular, the act 830 can involve training aresponse classification neural network based on the word-agnosticvectors for each word of the plurality of words from the plurality ofsentences, where the response classification neural network indicatesdistances between parts of speech associated with two word-agnosticvectors. In some embodiments, the act 830 also includes training theresponse classification neural network based on sequence learning thatlearns a word-agnostic vector for a word in a sentence of the pluralityof sentences based on one or more previous words in the sentence. Inaddition, in various embodiments, the trained response classificationneural network outputs a response classification label for an input wordwithin an input sentence.

In some embodiments, the act 830 further includes training the responseclassification neural network based on an attention mechanism thatcomputes a weighted representation of a sentence of the plurality ofsentences. In various embodiments, the trained response classificationneural network is a bi-directional long short-term memory recurrentneural network. Additionally, in one or more embodiments, the act 830includes pairing an opinion label with a target classification label inone or more sentences of the plurality of sentences as part of trainingthe response classification neural network.

As shown, the series of acts 800 also includes an act 840 of receivingan input sentence. In particular, the act 840 can involve receiving aninput sentence including one or more words. In one or more embodiments,the input sentence is an unclassified free-form text survey response(e.g., opinion text response).

In addition, the series of acts 800 includes an act 850 of determiningparts of speech for the input sentence. In particular, the act 850 caninvolve determining parts of speech for each word of the one or morewords in the input sentence. In one or more embodiments, the act 850includes automatically parsing the plurality of sentences to identifythe part of speech label associated with each word of the plurality ofwords, and assigning, to each word of the plurality of words, theidentified part of speech label corresponding to each word. In someembodiments, the number of parts of speech is 50 parts of speech.

As shown, the series of acts 800 further includes an act 860 ofdetermining response classification labels for the input sentence basedon the trained response classification neural network. In particular,the act 860 can involve determining, based on the one or more words andthe trained response classification neural network, a responseclassification label for each word of the plurality of words from theinput sentence. In one or more embodiments, the response classificationlabel includes one of a target classification label, an opinionclassification label, or a neither classification label.

As shown, the series of acts 800 also includes an act 870 of presentingthe input sentence to a user based on the determined responseclassification labels. In particular, the act 870 can involvepresenting, to a user, a target portion and an opinion portion of theinput sentence based on the response classification label determined foreach word of the plurality of words from the input sentence. In one ormore embodiments, the target portion includes words of the plurality ofwords having a target classification label. In some embodiments, theopinion portion includes words of the plurality of words having anopinion classification label.

The series of acts 800 can also include a number of additional acts. Inone or more embodiments, the series of acts 800 includes the act oftraining the response classification neural network as a firstbi-directional long short-term memory recurrent neural network. Inaddition, the series of acts 800 include the act of determining theresponse classification label for each word of the plurality of wordsfrom the input sentence based on a second bi-directional long short-termmemory recurrent neural network that provides a response classificationdetermination (e.g., response classification labels) for a previous wordin the input sentence as input to the first bi-directional longshort-term memory recurrent neural network when determining the responseclassification for a subsequent word in the input sentence.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., memory), and executes those instructions, thereby performing oneor more processes, including one or more of the processes describedherein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices), or vice versa.For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed by a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed by ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. As used herein, the term “cloud computing”refers to a model for enabling on-demand network access to a shared poolof configurable computing resources. For example, cloud computing can beemployed in the marketplace to offer ubiquitous and convenient on-demandaccess to the shared pool of configurable computing resources. Theshared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In addition, as used herein, the term “cloud-computingenvironment” refers to an environment in which cloud computing isemployed.

FIG. 9 illustrates a block diagram of an exemplary computing device 900that may be configured to perform one or more of the processes describedabove. One will appreciate that one or more computing devices, such asthe computing device 900 may represent the computing devices describedabove (e.g., server device 101, client devices 106, 108, 600, andcomputing device 700). In one or more embodiments, the computing device900 may be a mobile device (e.g., a mobile telephone, a smartphone, aPDA, a tablet, a laptop, a camera, a tracker, a watch, a wearabledevice, etc.). In some embodiments, the computing device 900 may be anon-mobile device (e.g., a desktop computer or another type of clientdevice). Further, the computing device 900 may be a server device thatincludes cloud-based processing and storage capabilities.

As shown in FIG. 9, the computing device 900 can include one or moreprocessor(s) 902, memory 904, a storage device 906, input/outputinterfaces 908 (or simply “I/O interfaces 908”), and a communicationinterface 910, which may be communicatively coupled by way of acommunication infrastructure (e.g., bus 912). While the computing device900 is shown in FIG. 9, the components illustrated in FIG. 9 are notintended to be limiting. Additional or alternative components may beused in other embodiments. Furthermore, in certain embodiments, thecomputing device 900 includes fewer components than those shown in FIG.9. Components of the computing device 900 shown in FIG. 9 will now bedescribed in additional detail.

In particular embodiments, the processor(s) 902 includes hardware forexecuting instructions, such as those making up a computer program. Asan example, and not by way of limitation, to execute instructions, theprocessor(s) 902 may retrieve (or fetch) the instructions from aninternal register, an internal cache, memory 904, or a storage device906 and decode and execute them.

The computing device 900 includes memory 904, which is coupled to theprocessor(s) 902. The memory 904 may be used for storing data, metadata,and programs for execution by the processor(s). The memory 904 mayinclude one or more of volatile and non-volatile memories, such asRandom-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-statedisk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of datastorage. The memory 904 may be internal or distributed memory.

The computing device 900 includes a storage device 906 includes storagefor storing data or instructions. As an example, and not by way oflimitation, the storage device 906 can include a non-transitory storagemedium described above. The storage device 906 may include a hard diskdrive (HDD), flash memory, a Universal Serial Bus (USB) drive or acombination these or other storage devices.

As shown, the computing device 900 includes one or more I/O interfaces908, which are provided to allow a user to provide input to (such asuser strokes), receive output from, and otherwise transfer data to andfrom the computing device 900. These I/O interfaces 908 may include amouse, keypad or a keyboard, a touch screen, camera, optical scanner,network interface, modem, other known I/O devices or a combination ofsuch I/O interfaces 908. The touch screen may be activated with a stylusor a finger.

The I/O interfaces 908 may include one or more devices for presentingoutput to a user, including, but not limited to, a graphics engine, adisplay (e.g., a display screen), one or more output drivers (e.g.,display drivers), one or more audio speakers, and one or more audiodrivers. In certain embodiments, I/O interfaces 908 are configured toprovide graphical data to a display for presentation to a user. Thegraphical data may be representative of one or more graphical userinterfaces and/or any other graphical content as may serve a particularimplementation.

The computing device 900 can further include a communication interface910. The communication interface 910 can include hardware, software, orboth. The communication interface 910 provides one or more interfacesfor communication (such as, for example, packet-based communication)between the computing device and one or more other computing devices orone or more networks. As an example, and not by way of limitation,communication interface 910 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI. The computingdevice 900 can further include a bus 912. The bus 912 can includehardware, software, or both that connects components of computing device900 to each other.

FIG. 10 illustrates an example network environment 1000 of a responseanalysis system 104, such as embodiments of the response analysis systemdescribed herein. The network environment 1000 includes the responseanalysis system 104 and a client device 1006 connected to each other bya network 1004. Although FIG. 10 illustrates a particular arrangement ofthe response analysis system 104, the client device 1006, and thenetwork 1004, one will appreciate that other arrangements of the networkenvironment 1000 are possible. For example, a client device of theclient device 1006 is directly connected to the response analysis system104. Moreover, this disclosure contemplates any suitable number ofclient systems, response analysis system s, and networks are possible.For instance, the network environment 1000 includes multiple clientsystems.

This disclosure contemplates any suitable network. As an example, one ormore portions of the network 1004 may include an ad hoc network, anintranet, an extranet, a VPN, a LAN, a wireless LAN, a WAN, a wirelessWAN, a MAN, a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, a safelightnetwork, or a combination of two or more of these. The term “network”may include one or more networks and may employ a variety of physicaland virtual links to connect multiple networks together.

In particular embodiments, the client device 1006 is an electronicdevice including hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by the clientsystem. As an example, the client device 1006 includes any of thecomputing devices discussed above. The client device 1006 may enable auser at the client device 1006 to access the network 1004. Further, theclient device 1006 may enable a user to communicate with other users atother client systems.

In some embodiments, the client device 1006 may include a web browser,such as and may have one or more add-ons, plug-ins, or other extensions.The client device 1006 may render a web page based on the HTML filesfrom the server for presentation to the user. For example, the clientdevice 1006 renders the graphical user interface described above.

In one or more embodiments, the response analysis system 104 includes avariety of servers, sub-systems, programs, modules, logs, and datastores. In some embodiments, response analysis system 104 includes oneor more of the following: a web server, action logger, API-requestserver, relevance-and-ranking engine, content-object classifier,notification controller, action log, third-party-content-object-exposurelog, inference module, authorization/privacy server, search module,user-targeting module, user-interface module, user-profile store,connection store, third-party content store, or location store. Theresponse analysis system 104 may also include suitable components suchas network interfaces, security mechanisms, load balancers, failoverservers, management-and-network-operations consoles, other suitablecomponents, or any suitable combination thereof.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with fewer or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel to one another or inparallel to different instances of the same or similar steps/acts. Thescope of the invention is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes that come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

What is claimed is:
 1. A system comprising: at least one processor; andat least one non-transitory computer-readable storage medium storinginstructions that, when executed by the at least one processor, causethe system to: generate word-agnostic vectors for each word of aplurality of words from a plurality of sentences, wherein theword-agnostic vectors for each word are based on: a part of speech labelassigned to each word of the plurality of words; and a responseclassification label assigned to each word of the plurality of words;and train a response classification neural network based on theword-agnostic vectors for each word of the plurality of words from theplurality of sentences, wherein the response classification neuralnetwork associates parts of speech with response classification labels,and wherein the trained response classification neural network outputs aresponse classification label for an input word within an inputsentence.
 2. The system of claim 1, wherein the word-agnostic vectorsomit semantic word information corresponding to the plurality of wordsfrom each of the plurality of sentences.
 3. The system of claim 1,wherein the response classification label comprises one of a targetclassification label, an opinion classification label, or a neitherclassification label.
 4. The system of claim 1, wherein theinstructions, when executed by the at least one processor, cause thesystem to train the response classification neural network using abi-directional long short-term memory recurrent neural network.
 5. Thesystem of claim 1, wherein: the plurality of sentences represent is asmall-data training dataset comprising 2,500 or fewer sentences, and thepart of speech label assigned to each word comprises one of 50 parts ofspeech.
 6. The system of claim 1, further comprising instructions that,when executed by the at least one processor, cause the system to:automatically parse the plurality of sentences to identify the part ofspeech label associated with each word of the plurality of words; andassign, to each word of the plurality of words, the identified part ofspeech label corresponding to each word.
 7. The system of claim 1,wherein the instructions, when executed by the at least one processor,cause the system to generate the word-agnostic vectors for each word ofthe plurality of words is further based on a dependency label assignedto each word of the plurality of words.
 8. The system of claim 1,wherein the instructions, when executed by the at least one processor,cause the system to generate the word-agnostic vectors for each word ofthe plurality of words is further based on transfer information thatclassifies one or more particular words not included in a trainingdataset with a defined part of speech label.
 9. The system of claim 1,wherein the instructions, when executed by the at least one processor,cause the system to train the response classification neural network isfurther based on sequence learning that learns a word-agnostic vectorfor a word in a sentence of the plurality of sentences based on one ormore previous words in the sentence.
 10. The system of claim 1, whereinthe instructions, when executed by the at least one processor, cause thesystem to train the response classification neural network is furtherbased on an attention mechanism that computes a weighted representationof a sentence of the plurality of sentences.
 11. A non-transitorycomputer-readable medium storing instructions that, when executed by atleast one processor, cause a computer system to: receive an inputsentence comprising a plurality of words; determine a part of speech foreach word of the plurality of words; determine, by a responseclassification neural network, a response classification label for eachword of the plurality of words based on the determined parts of speechfor each word of the plurality of words, wherein the responseclassification neural network associates word-agnostic vectorscomprising parts of speech; and present, to a user, a target portion andan opinion portion of the input sentence based on the responseclassification label determined for each word of the plurality of wordsfrom the input sentence.
 12. The non-transitory computer-readable mediumof claim 11, wherein the input sentence is a free-form text surveyresponse.
 13. The non-transitory computer-readable medium of claim 11,wherein the target portion comprises at least one word of the pluralityof words corresponding to a target classification label, and wherein theopinion portion comprises at least one word of the plurality of wordscorresponding to an opinion classification label.
 14. The non-transitorycomputer-readable medium of claim 11, wherein the responseclassification neural network is a first bi-directional long short-termmemory recurrent neural network.
 15. The non-transitorycomputer-readable medium of claim 14, wherein the instructions, whenexecuted by the at least one processor, cause the computer system todetermine the response classification label for each word of theplurality of words from the input sentence based on a secondbi-directional long short-term memory recurrent neural network thatprovides a response classification determination for a previous word inthe input sentence as input to the first bi-directional long short-termmemory recurrent neural network when determining the responseclassification for a subsequent word in the input sentence.
 16. A methodcomprising: receiving a training dataset comprising a plurality ofsentences, each sentence associated with a plurality of words;generating word-agnostic vectors for each word of the plurality of wordsassociated with each sentence of the plurality of sentences based on: apart of speech label assigned to each word of the plurality of words; adependency label assigned to each word of the plurality of words; and aresponse classification label assigned to each word of the plurality ofwords; training a response classification neural network based on theword-agnostic vectors for each word of the plurality of words associatedwith each sentence of the plurality of sentences; determining parts ofspeech for one or more words in an input sentence; based on the parts ofspeech for the one or more words, determining, by the trained responseclassification neural network, a response classification label for eachof the one or more words from the input sentence; and providing a targetportion and an opinion portion of the input sentence based on theresponse classification label determined for each word of the one ormore words from the input sentence.
 17. The method of claim 16, whereinthe word-agnostic vectors omit semantic word information correspondingto the plurality of words from each of the plurality of sentences. 18.The method of claim 16, wherein the response classification labeldetermined for each word of the one or more words from the inputsentence comprises one of a target classification label, an opinionclassification label, or a neither classification label.
 19. The methodof claim 18, wherein the target portion comprises words of the one ormore words having a target classification label, and wherein the opinionportion comprises words of the one or more words having an opinionclassification label.
 20. The method of claim 16, wherein training theresponse classification neural network comprising pairing an opinionlabel with a target classification label in one or more sentences of theplurality of sentences.